{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bf733a38-db84-4363-89e2-de6735c37230",
   "metadata": {},
   "source": [
    "# DeepInfra\n",
    "\n",
    "[DeepInfra](https://deepinfra.com/?utm_source=langchain) is a serverless inference as a service that provides access to a [variety of LLMs](https://deepinfra.com/models?utm_source=langchain) and [embeddings models](https://deepinfra.com/models?type=embeddings&utm_source=langchain). This notebook goes over how to use LangChain with DeepInfra for chat models.\n",
    "\n",
    "## Set the Environment API Key\n",
    "Make sure to get your API key from DeepInfra. You have to [Login](https://deepinfra.com/login?from=%2Fdash) and get a new token.\n",
    "\n",
    "You are given a 1 hour free of serverless GPU compute to test different models. (see [here](https://github.com/deepinfra/deepctl#deepctl))\n",
    "You can print your token with `deepctl auth token`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get a new token: https://deepinfra.com/login?from=%2Fdash\n",
    "\n",
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "from langchain_community.chat_models import ChatDeepInfra\n",
    "from langchain_core.messages import HumanMessage\n",
    "\n",
    "DEEPINFRA_API_TOKEN = getpass()\n",
    "\n",
    "# or pass deepinfra_api_token parameter to the ChatDeepInfra constructor\n",
    "os.environ[\"DEEPINFRA_API_TOKEN\"] = DEEPINFRA_API_TOKEN\n",
    "\n",
    "chat = ChatDeepInfra(model=\"meta-llama/Llama-2-7b-chat-hf\")\n",
    "\n",
    "messages = [\n",
    "    HumanMessage(\n",
    "        content=\"Translate this sentence from English to French. I love programming.\"\n",
    "    )\n",
    "]\n",
    "chat.invoke(messages)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c",
   "metadata": {},
   "source": [
    "## `ChatDeepInfra` also supports async and streaming functionality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93a21c5c-6ef9-4688-be60-b2e1f94842fb",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_core.callbacks import StreamingStdOutCallbackHandler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "await chat.agenerate([messages])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "025be980-e50d-4a68-93dc-c9c7b500ce34",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "chat = ChatDeepInfra(\n",
    "    streaming=True,\n",
    "    verbose=True,\n",
    "    callbacks=[StreamingStdOutCallbackHandler()],\n",
    ")\n",
    "chat.invoke(messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "466c3cb41ace1410",
   "metadata": {},
   "source": [
    "# Tool Calling\n",
    "\n",
    "DeepInfra currently supports only invoke and async invoke tool calling.\n",
    "\n",
    "For a complete list of models that support tool calling, please refer to our [tool calling documentation](https://deepinfra.com/docs/advanced/function_calling)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ddc4f4299763651c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "from dotenv import find_dotenv, load_dotenv\n",
    "from langchain_community.chat_models import ChatDeepInfra\n",
    "from langchain_core.messages import HumanMessage\n",
    "from langchain_core.pydantic_v1 import BaseModel\n",
    "from langchain_core.tools import tool\n",
    "\n",
    "model_name = \"meta-llama/Meta-Llama-3-70B-Instruct\"\n",
    "\n",
    "_ = load_dotenv(find_dotenv())\n",
    "\n",
    "\n",
    "# Langchain tool\n",
    "@tool\n",
    "def foo(something):\n",
    "    \"\"\"\n",
    "    Called when foo\n",
    "    \"\"\"\n",
    "    pass\n",
    "\n",
    "\n",
    "# Pydantic class\n",
    "class Bar(BaseModel):\n",
    "    \"\"\"\n",
    "    Called when Bar\n",
    "    \"\"\"\n",
    "\n",
    "    pass\n",
    "\n",
    "\n",
    "llm = ChatDeepInfra(model=model_name)\n",
    "tools = [foo, Bar]\n",
    "llm_with_tools = llm.bind_tools(tools)\n",
    "messages = [\n",
    "    HumanMessage(\"Foo and bar, please.\"),\n",
    "]\n",
    "\n",
    "response = llm_with_tools.invoke(messages)\n",
    "print(response.tool_calls)\n",
    "# [{'name': 'foo', 'args': {'something': None}, 'id': 'call_Mi4N4wAtW89OlbizFE1aDxDj'}, {'name': 'Bar', 'args': {}, 'id': 'call_daiE0mW454j2O1KVbmET4s2r'}]\n",
    "\n",
    "\n",
    "async def call_ainvoke():\n",
    "    result = await llm_with_tools.ainvoke(messages)\n",
    "    print(result.tool_calls)\n",
    "\n",
    "\n",
    "# Async call\n",
    "asyncio.run(call_ainvoke())\n",
    "# [{'name': 'foo', 'args': {'something': None}, 'id': 'call_ZH7FetmgSot4LHcMU6CEb8tI'}, {'name': 'Bar', 'args': {}, 'id': 'call_2MQhDifAJVoijZEvH8PeFSVB'}]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
